Fall detection and reporting technology

ABSTRACT

Fall detection and reporting technology, in which output from at least one sensor configured to sense, in a room of a building, activity associated with a patient falling is monitored and a determination is made to capture one or more images of the room based on the monitoring. An image of the room is captured with a camera positioned to include the patient within a field of view of the camera and the captured image of the room is analyzed to detect a state of the patient at a time of capturing the image. A potential fall event for the patient is determined based on the detected state of the patient and a message indicating the potential fall event for the patient is sent based on the determination of the potential fall event for the patient. Techniques are also described for fall detection and reporting using an on-body sensing device.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of U.S. ProvisionalApplication No. 61/471,495, filed Apr. 4, 2011, which is incorporatedherein by reference in its entirety for all purposes.

TECHNICAL FIELD

This disclosure relates to fall detection and reporting technology.

BACKGROUND

Falls are a public health concern and cause for institutionalization inthe senescent population, for whom they disproportionately affect.Loosely defined as an unintentional and uncontrolled movement towardsthe ground or lower level, a fall can have debilitating and sometimesfatal consequences. Although falls increase rates of morbidity andmortality, earlier detection and reporting of such events can improveoutcomes.

Practical, early detection and reporting of falls has been an elusivegoal. Efforts to detect falls have classically employed wearabletechnologies to capture user input (e.g., panic button press) or tocharacterize and classify movements and postures. Although thesetechnologies demonstrate reasonable utility in ideal conditions, usernon-compliance and fall-related incapacitation reduce general efficacyin application. Furthermore, inability to verify incidence of detectedfalls (e.g., both true and false) leads to inaccurate fall reporting andundesirable handling of potential fall events.

SUMMARY

Techniques are described for fall detection and reporting technology. Inone aspect, a method includes monitoring output from at least one sensorconfigured to sense, in a room of a building, activity associated with apatient falling and, based on the monitoring of output from the at leastone sensor, determining to capture one or more images of the room. Themethod also includes capturing, with a camera positioned to include thepatient within a field of view of the camera, an image of the room andanalyzing the captured image of the room to detect a state of thepatient at a time of capturing the image. The method further includesdetermining, based on the detected state of the patient, a potentialfall event for the patient and, based on the determination of thepotential fall event for the patient, sending, by a communicationdevice, a message indicating the potential fall event for the patient.

Implementations may include one or more of the following features. Forexample, the at least one sensor configured to sense activity associatedwith the patient falling may be an on-body sensor configured to detectan impact and determine a change in an orientation of the patient. Inthis example, the method may include receiving data indicating adetected change in an orientation of the patient and an amount of theorientation change, receiving data indicating a detected impact and aseverity of the detected impact, and determining, based on the receivedamount of orientation change and the received data indicating theseverity of the impact of the patient, a threshold for inactivity of thepatient. The method also may include determining, based on output fromthe on-body sensor, that the patient has been inactive for a period oftime greater than the determined threshold and determining to captureone or more images of the room based on the determination that thepatient has been inactive for a period of time greater than thedetermined threshold.

In addition, the at least one sensor configured to sense activityassociated with the patient falling may be a button located in the roomat a position that permits the patient to press the button after a falland the method may include determining that the button has been pressed.The at least one sensor configured to sense activity associated with thepatient falling may be a sensor configured to determine a presence ofthe patient in the room and the method may include receiving, from thesensor configured to determine the presence of the patient in the room,a signal indicating that the patient is present in the room and, after athreshold period of time, determining that the patient has not left theroom and that no further signals have been received from the sensorconfigured to determine the presence of the patient in the room. Themethod may include determining to capture one or more images of the roombased on determining that the patient has not left the room and that nofurther signals have been received from the sensor configured todetermine the presence of the patient in the room.

In some examples, the method may include performing image foregroundsegmentation on the captured image to create a segmented image,performing template matching on the segmented image to identify a humanshape in the segmented image, and calculating a position and anorientation associated with the identified human shape in the segmentedimage. In these examples, the method may include determining a potentialfall event for the patient based on the calculated position and thecalculated orientation. Further, in these examples, the method mayinclude monitoring successive image and sensor data after calculatingthe position and the orientation, comparing the successive image andsensor data with prior image and sensor data, determining an activitylevel of the patient based on the comparison of the successive image andsensor data with the prior image and sensor data, classifying thepotential fall event based on the determined activity level of thepatient, and handling reporting for the potential fall event based onthe classification of the potential fall event.

In some implementations, the method may include analyzing the monitoredoutput from the at least one sensor over a period of time to determineactivities of the patient over the period of time and accessinginformation indicative of expected activities of the patient over theperiod of time. In these implementations, the method may includecomparing the determined activities of the patient over the period oftime to the expected activities of the patient over the period of timeand, based on the comparison revealing that the determined activities ofthe patient over the period of time do not match the expected activitiesof the patient over the period of time, determining a level of fall riskassociated with the patient.

The method may include determining that the level of fall riskassociated with the patient exceeds a threshold and, based on thedetermination that the level of fall risk associated with the patientexceeds the threshold, sending a message to a monitoring server that islocated remotely from the building. The method also may includedetermining that the level of fall risk associated with the patientexceeds a threshold and, based on the determination that the level offall risk associated with the patient exceeds the threshold,automatically performing one or more operations to reduce the level offall risk associated with the patient.

In some examples, the method may include sending, to the patient, themessage indicating the potential fall event and providing the patientwith an opportunity to cancel the potential fall event. In theseexamples, the method may include determining that the patient has notcancelled the potential fall event within a threshold period of timeand, based on determining that the patient has not cancelled thepotential fall event within the threshold period of time, sending amessage to a monitoring server indicating the potential fall event.Further, in these examples, the method may include receiving, from thepatient, an indication to cancel the potential fall event and, based onreceiving the indication to cancel the potential fall event, determiningan overall activity of the patient between detecting the potential fallevent and receiving the indication to cancel the potential fall event.

In addition, the method may include determining that the overallactivity of the patient is above a threshold of activity and, based ondetermining that the overall activity of the patient is above thethreshold of activity, signaling that the potential fall event wasdetection of a false fall. The method also may include determining thatthe overall activity of the patient is below a threshold of activityand, based on determining that the overall activity of the patient isbelow the threshold of activity, determining an orientation of thepatient. The method further may include determining that the determinedorientation of the patient is upright and, based on determining that thedetermined orientation of the patient is upright, signaling that thepotential fall event was detection of a minor fall.

In some implementations, the method may include determining that thedetermined orientation of the patient is not upright and, based ondetermining that the determined orientation of the patient is notupright, sending another message to the patient that provides thepatient with another opportunity to cancel the potential fall event. Inthese implementations, the method may include determining that thepatient has not cancelled the potential fall event within a thresholdperiod of time after sending another message to the patient thatprovides the patient with another opportunity to cancel the potentialfall event and, based on determining that the patient has not cancelledthe potential fall event within the threshold period of time aftersending another message to the patient that provides the patient withanother opportunity to cancel the potential fall event, sending amessage to a monitoring server indicating the potential fall event.Also, in these implementations, the method may include after sendinganother message to the patient that provides the patient with anotheropportunity to cancel the potential fall event, receiving, from thepatient, an indication to cancel the potential fall event and, based onreceiving the indication to cancel the potential fall event, signalingthat the potential fall event was a cancelled fall event.

Implementations of the described techniques may include hardware, amethod or process implemented at least partially in hardware, or acomputer-readable storage medium encoded with executable instructionsthat, when executed by a processor, perform operations.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features will beapparent from the description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIGS. 1, 2, and 4 to 6 illustrate example systems.

FIGS. 3, 7, 8, 10, and 11 are flow charts illustrating exampleprocesses.

FIG. 9 is illustrates example fall detection criteria.

FIG. 12 is a diagram illustrating fall detection examples.

DETAILED DESCRIPTION

Techniques are described for addressing the aforementioned falldetection and reporting challenges. For example, a monitoring system ina premise performs fall detection and reporting operations based onoutput from a sensor (e.g., an image sensor). When the monitoring systemdetects that a person has fallen in the premise, actions are taken toassist the fallen person.

FIG. 1 illustrates an image sensing device 110 that may be installedwithin a monitored home or facility. The image sensing device 110combines multi-modal sensing (e.g., passive infrared motion sensor,triaxial inertial sensor, illumination sensor), an infrared illuminationsource, camera, processor, memory, battery, and input/outputcapabilities. The image sensing device 110 detects events indicative ofpotential falls proximal to its installation location. A plurality ofimage sensing devices can be installed throughout a home or facility,and used in conjunction with other sensors, to increase the falldetection coverage area and provide specific location information forfall reporting and response.

The image sensing device 110 includes a processor 111, a memory 112, acamera 113, an illumination source 114, a motion sensor 115, anillumination sensor 116, a battery 117, and an input/output port 118.The processor 111 controls operations of the image sensing device 110and may be any suitable processor. The memory 112 stores instructionsthat are executed by the processor 111 and also stores images capturedby the camera 113. The memory 112 may be any type of memory that iscapable storing data and may include a combination of multiple, memoryunits. For example, the memory 112 may be a Flash memory component thatstores both instructions that are executed by the processor and imagescaptured by the camera 113.

The camera 113 captures images of an area proximate to where the imagesensing device is located. For instance, the camera 113 may be placed atan upper corner of a room in a building and, in this instance, thecamera 113 captures images of the room. The camera 113 may be avideo/photographic camera or other type of optical sensing deviceconfigured to capture images. In some implementations, the camera 113 isa CMOS camera sensor (or other CCD sensor) that captures images atvarious, different resolutions (e.g., low and/or high resolutions). Forinstance, the CMOS camera sensor may capture 640×480 pixels (e.g., VGAresolution) or higher resolutions. The camera 113 also may capture alower resolution image (e.g., Quarter VGA=QVGA=320×240 pixels).

The illumination source 114 may be any source of illumination thatimproves capturing of images in a dark area. For example, theillumination source 114 may include one or more Infra Red LEDs that emitInfra Red light over an area within a field of view of the camera 113 toilluminate objects within the area. The processor 111 may control theillumination source 114 to emit light when the illumination sensor 116detects a level of light that is below a threshold level.

The motion sensor 115 may be Passive Infra Red (PIR) motion sensor, amicrowave motion sensor, or any type of sensor that detects motion in anarea corresponding to a field of view of the camera 113. The processor111 may monitor output of the motion sensor 115 and trigger the camera113 to capture images in response to the motion sensor 115 detectingmotion in the area corresponding to the field of view of the camera 113.

The battery 117 is the power source of the image sensing device 110 andmay be any type of battery capable of delivering power to the imagesensing device 110. The battery 117 may have a relatively small size andmay be a standard type of battery available for purchase at retailstores. The battery 117 may be located in a compartment that is easilyaccessible to a user of the image sensing device 110 to facilitatechanging of the battery 117, which may occur relatively frequently(e.g., every couple of months) depending on the power consumption andimage capture settings of the image sensing device 110.

The input/output port 118 is a communication interface through which theimage sensing device may send and receive wireless communications. Theinput/output port 118 may, using a short range wireless protocol (e.g.,Bluetooth, Z-Wave, ZigBee, local wireless 900 MHz communication band,etc.), receive and send short range wireless communications with otherdevices. The input/output port 118 may include a “normally open” or“normally closed” digital input that can trigger capture of images usingthe camera 113.

To reduce processing power needed and to conserve battery life, theprocessor 111 may control components of the image sensing device 110 toperiodically enter sleep mode operation. For example, the processor 111may awaken every second to determine whether any communications havebeen received at the input/output port 118. If no communications havebeen received, the processor 111 may place itself and other components(e.g., the memory 112, the camera 113, etc.) in a sleep mode for anothersecond before awaking again to determine whether any communications havebeen received at the input/output port 118. The processor 111 also mayawaken from a sleep mode state based on output from the motion sensor115 indicating that motion has been detected and/or based on output froman “inertial sensor” that detects impacts to the image sensing device110.

FIG. 2 illustrates an example of an electronic system 200 configured toprovide fall detection and reporting. The system 200 includes the imagesensing device 110, a gateway 120, one or more remote monitoring servers130, one or more user devices 140, and a central monitoring station 150.The image sensing device 110 is a relatively small and affordable unitthat captures still images of an area that corresponds to a location ofthe image sensing device. Because the image sensing device 110 isrelatively small, runs off of battery power, and communicates via awireless communication protocol, the image sensing device 110 may beeasily placed at any location within a monitored property (or justoutside of a monitored property) to provide image surveillance of anarea of the monitored property (or an area just outside of the monitoredproperty).

The gateway 120 is a communication device configured to exchange shortrange wireless communications with the image sensing device 110 and longrange wireless or wired communications with the remote monitoring server130 over the network 135. Because the gateway 120 exchanges short rangewireless communications with the image sensing device 110, the gateway120 is positioned nearby the image sensing device 110. As shown in FIG.2, the gateway 120 and the image sensing device 110 are both locatedwithin a monitored property that is remote (and may be very far awayfrom) the remote monitoring server 130.

In some examples, the gateway 120 may include a wireless communicationdevice configured to exchange long range communications over a wirelessdata channel. In this example, the gateway 120 may transmit header dataand image data over a wireless data channel. The gateway 120 may includeone or more of a GSM module, a radio modem, cellular transmissionmodule, or any type of module configured to exchange communications inone of the following formats: GSM or GPRS, CDMA, EDGE or EGPRS, EV-DO orEVDO, or UMTS.

The gateway 120 includes a buffer memory 122 that stores image datacaptured by the image sensing device 110. The buffer memory 122 maytemporarily store image data captured by the image sensing device 110 todelay a decision of whether the image data (or a subset of the imagedata) is worthwhile to send to the remote monitoring server 130. Thebuffer memory 122 may be larger than the memory 112 of the image sensingdevice 110 and, because the gateway 120 operates using an AC powersource, using the buffer memory 122 to store images captured by theimage sensing device 110 may be more efficient. The gateway 120 also mayinclude a display with which the stored images may be displayed to auser.

The long range wireless network 135 enables wireless communicationbetween the gateway 120 and the remote monitoring server 130. The longrange wireless network 135 may be any type of cellular network and maysupport any one or more of the following protocols: GSM or GPRS, CDMA,EDGE or EGPRS, EV-DO or EVDO, or UMTS. It may be relatively expensive totransmit data over the long range wireless network 135 and, therefore,the image sensing device 110 and the gateway 120 may be selective in theimage data transmitted to the remote monitoring server 130.

The remote monitoring server 130 receives image data from the gateway120 over the long range wireless or wired network 135. The remotemonitoring server 130 stores the received image data and makes the imagedata available to one or more user devices 140 and/or the centralmonitoring station 150 over the IP-based network 145. For instance, theremote monitoring server 130 may make the image data available to theone or more user devices 140 and/or the central monitoring station 150at a website accessible by the one or more user devices 140 and/or thecentral monitoring station 150 over the Internet. The remote monitoringserver 130 also may make the image data available to the one or moreuser devices 140 and/or the central monitoring station 150 in anelectronic message, such as an electronic mail message.

In some implementations, the remote monitoring server 130 receives theimage data from the gateway 120 as a reference image and a series ofdifferential images that indicate the difference between thecorresponding image and the reference image. In these implementations,header information sent with the image data indicates which images arereference images, which images are differential images, and whichreference image each differential image corresponds to. The remotemonitoring server 130 processes the reference image and the differentialimages and converts each image into a standard image format, such asJPEG. The remote monitoring server 130 then stores the converted imagesin a database or a file system and makes the converted images availableto the one or more user devices 140 and/or the central monitoringstation 150.

The central monitoring station 150 includes an electronic device (e.g.,a server) configured to provide alarm monitoring service by exchangingcommunications with the remote monitoring server 130 over the network145. For example, the central monitoring station 150 may be configuredto monitor alarm events generated by a monitoring or alarm system thatmonitors the home or facility where the image sensing device 110 islocated. In this example, the central monitoring station 150 mayexchange communications with the remote monitoring server 130 to receiveinformation regarding alarm events detected by the monitoring or alarmsystem. The central monitoring station 150 also may receive informationregarding alarm events from the one or more user devices 140. Thecentral monitoring station 150 may receive images captured by the imagesensing device 110 to enable verification of potential fall events.

The central monitoring station 150 may be connected to multipleterminals. The terminals may be used by operators to process alarmevents. For example, the central monitoring station 150 may route alarmdata to the terminals to enable an operator to process the alarm data.The terminals may include general-purpose computers (e.g., desktoppersonal computers, workstations, or laptop computers) that areconfigured to receive alarm data from a server in the central monitoringstation 150 and render a display of information based on the alarm data.For example, the central monitoring station 150 may receive alarm dataand route the alarm data to a terminal for processing by an operatorassociated with the terminal. The terminal may render a display to theoperator that includes information associated with the alarm event(e.g., the name of the user of the alarm system, the address of thebuilding the alarm system is monitoring, the type of alarm event, imagesof fall events taken of the image sensing device 110, etc.) and theoperator may handle the alarm event based on the displayed information.

The one or more user devices 140 include devices that host userinterfaces. For instance, the user devices 140 may include a mobiledevice that hosts one or more native applications (e.g., a falldetection and reporting application). The user devices 140 may include acellular phone or a non-cellular locally networked device with adisplay. The user devices 140 may include a smart phone, a tablet PC, apersonal digital assistant (“PDA”), or any other portable deviceconfigured to communicate over a network and display information. Forexample, implementations may also include Blackberry-type devices (e.g.,as provided by Research in Motion), electronic organizers, iPhone-typedevices (e.g., as provided by Apple), iPod devices (e.g., as provided byApple) or other portable music players, other communication devices, andhandheld or portable electronic devices for gaming, communications,and/or data organization. The user devices 140 may perform functionsunrelated to the monitoring system, such as placing personal telephonecalls, playing music, playing video, displaying pictures, browsing theInternet, maintaining an electronic calendar, etc.

The user devices 140 may include a native fall detection and reportingapplication. The native fall detection and reporting application refersto a software/firmware program running on the corresponding mobiledevice that enables the user interface and features describedthroughout. The user devices 140 may load or install the native falldetection and reporting application based on data received over anetwork or data received from local media. The native fall detection andreporting application runs on mobile device platforms, such as iPhone,iPod touch, Blackberry, Google Android, Windows Mobile, etc. The nativefall detection and reporting application enables the user devices 140 toreceive and process image and sensor data from the monitoring system.

The user devices 140 also may include a general-purpose computer (e.g.,a desktop personal computer, a workstation, or a laptop computer) thatis configured to communicate with the remote monitoring server 130 overthe network 145. The user devices 140 may be configured to display afall detection and reporting user interface that is generated by theuser devices 140 or generated by the remote monitoring server 130. Forexample, the user devices 140 may be configured to display a userinterface (e.g., a web page) provided by the remote monitoring server130 that enables a user to perceive images captured by the image sensingdevice 110 and/or reports related to the monitoring system.

The system 200 further includes one or more trigger sources 128. Thetrigger sources 128 may include devices that assist in detecting fallevents. For example, the trigger sources 128 may include contact orpressure sensors that are positioned at a lower part of a building(e.g., at or near the floor). In this example, when a person falls, theperson may touch one of the trigger sources 128 to alert the system 200to the fall. In this regard, the system 200 may use output of thetrigger sources 128 to identify a possible fall location and begincapturing and processing images near that location to determine whetherthe trigger relates to an actual fall event or a false alarm, such asinadvertent contact with a trigger source.

In some examples, the system 200 may include inertial sensors (e.g.,accelerometers) to detect an impact potentially generated from a fall.In these examples, when a person falls, the inertial sensors may detectan impact and the system 200 may use the detected impact to infer apotential fall. In this regard, the system 200 may use output of theinertial sensors to identify a possible fall location and begincapturing and processing images near that location to determine whetherthe detected impact relates to an actual fall event or a false alarm,such as dropping of an object that resulted in the detected impact.

In some implementations, the image sensing device 110 and the gateway120 may be part of a home or facility monitoring system (e.g., a homesecurity system). In these implementations, the home or facilitymonitoring system may sense many types of events or activitiesassociated with the home or facility and the sensed events or activitiesmay be leveraged in performing fall detection and reporting features.The home or facility monitoring system may include a controller thatcommunicates with the gateway 120. The controller may be configured tocontrol the home or facility monitoring system (e.g., a home alarm orsecurity system). In some examples, the controller may include aprocessor or other control circuitry configured to execute instructionsof a program that controls operation of an alarm system. In theseexamples, the controller may be configured to receive input fromsensors, detectors, or other devices included in the home or facilitymonitoring system and control operations of devices included in the homeor facility monitoring system or other household devices (e.g., athermostat, an appliance, lights, etc.).

The home or facility monitoring system also includes one or more sensorsor detectors. For example, the home or facility monitoring system mayinclude multiple sensors, including a contact sensor, a motion sensor, aglass break sensor, or any other type of sensor included in an alarmsystem or security system. The sensors also may include an environmentalsensor, such as a temperature sensor, a water sensor, a rain sensor, awind sensor, a light sensor, a smoke detector, a carbon monoxidedetector, an air quality sensor, etc. The sensors further may include ahealth monitoring sensor, such as a prescription bottle sensor thatmonitors taking of prescriptions, a blood pressure sensor, a blood sugarsensor, a bed mat configured to sense presence of liquid (e.g., bodilyfluids) on the bed mat, bathroom usage sensors, food consumptionsensors, etc. In some examples, the sensors 120 may include aradio-frequency identification (RFID) sensor that identifies aparticular article that includes a pre-assigned RFID tag.

The system 200 shown in FIG. 2 may be used for the two example processes300 and 400 of fall detection and reporting described with respect toFIGS. 3 and 4. The example processes 300 and 400 are independent;however, they can be staged so that first level fall detection triggersfurther (e.g., second level) analysis and classification of potentialfall events. Both processes 300 and 400 have multiple steps, although asubset of steps may be employed to still meet practical requirements offall detection and reporting.

FIG. 3 illustrates an example process 300 for fall detection andreporting. The operations of the example process 300 are describedgenerally as being performed by the system 200. The operations of theexample process 300 may be performed by one of the components of thesystem 200 (e.g., the image sensing device 110, the gateway 120, theremote monitoring server 130, etc.) or may be performed by anycombination of the components of the system 200. In someimplementations, operations of the example process 300 may be performedby one or more processors included in one or more electronic devices.

In general, the process 300 enables fall detection and reporting basedon room occupancy analysis. The system 200 detects room occupancy (310).For example, movement events may be detected by the image sensing deviceor other external sensors (e.g., perceived motion by passive infraredmotion sensor of the image sensing devices, door openings and closingsdetected by door/window contact sensors of a home security system). Inthis example, the movement events signal possible human entrance into aroom where the image sensing device is located and are used to detectroom occupancy. The system 200 may capture camera image(s) and analyzethe camera image(s) to verify that the room is occupied.

After detecting room occupancy, the system 200 detects a lack of roomvacation (320). For example, the system 200 monitors output of the imagesensing device or other external sensors for movement events in theoccupied room and other rooms in the property. In this example, thesystem 200 detects successive movement events based on sensors of theimage device or other external sensors (even in other rooms). Thesuccessive movement events signal human vacation of the room and thesystem 200 analyzes the successive movement events to determine whetherthe room has been vacated. For instance, the system 200 may determinethat the room has been vacated when no successive movement events aredetected in the room and successive movement events are detected inother rooms of the property. The system 200 may determine that the roomhas not been vacated when successive movement events are detected in theroom and/or no successive movement events are detected in other rooms ofthe property. Based on a determination that the room has been vacated,the system 200 ceases further analysis and does not perform falldetection processing for the room until the room is detected as beingoccupied again.

Based on a determination that the room remains occupied, the system 200captures one or more images for analysis and/or reporting (330). Forinstance, if sensors indicate that the room remains occupied, butfurther movement has ceased over a prescribed and configurable intervalof time, the system 200 initiates image capture for reporting, furtherassessment, and/or validation of the possible fall event.

FIG. 4 illustrates another example of an electronic system 400configured to provide fall detection and reporting. The system 400includes one or more passive sensors 410, one or more assistance devices420, one or more imaging sensors 430, one or more user interface devices440, a gateway device 450, one or more remote servers 460, and amonitoring center 470. The one or more user interface devices 440, thegateway device 450, the one or more remote servers 460, and themonitoring center 470 may exchange communications over a communicationnetwork 480.

Passive sensors 410 may be employed to measure activity or inactivitywithin a monitored residence. The activity or inactivity can beassociated with a fall (e.g., impact, period of inactivity, location,time, etc.) or it can measure aspects of behavior related to fall risk(e.g., general activity level, sleeping, eating, bathroom use,medication use, gait speed, etc.). The behavior profiling can help topromote fall risk reduction via automated assistance devices 420 orthrough behavior change suggestions via user interface device(s) 440.

Assistance devices 420 are capable of performing automated tasks basedon inputs from sensors 410, a gateway device 450, user interfacedevice(s) 440, or remote servers 460. Assistance devices 420 can beprogrammed to respond based on rules specified by users, by caregivers,or by default. For example, a light can be illuminated in response to abed sensor being vacated during the evening. Assistance devices 420 canalso report their state to other devices, systems, or stakeholders.

Imaging sensors 430 (e.g., still frame or video) are capable ofdetecting possible falls. Furthermore, imaging sensors 430 can forwardimages of possible falls to remote servers 460, caregivers, ormonitoring centers 470 for automated or human verification. Imagingsensors 430 may also have other modes of sensing (e.g., motion,acceleration, etc.) to trigger or augment native imaging and sensingcapabilities. For example, impact sensed by the image sensor 430 couldbe used to trigger image capture. Captured images, sensed data, or otherinformation (e.g., location, time, etc.) may be communicated to otherdevices, systems, or stakeholders.

In some implementations, the image sensing device 110 described abovewith respect to FIG. 1 may be used as the imaging sensors 430. The imagesensing device 110 may be installed within a monitored home or facility.The device 110 combines multi-modal sensing (e.g., passive infraredmotion sensor, triaxial inertial sensor, illumination sensor), aninfrared illumination source, camera, processor, memory, battery,input/output, and radio (e.g., via input/output) capabilities. Thedevice 110 detects events indicative of potential falls proximal to itsinstallation location. A plurality of devices 110 may be installedthroughout a home or facility, and used in conjunction with othersensors, to increase the fall detection coverage area and providespecific location information for fall reporting and response.

A user interface device 440 may be used to communicate information to orgather information from a user about activity related to fallprevention, fall detection, or daily living. Possible physicalincarnations of user interface devices 440 may include light or audiosources, displays, push buttons, or mobile devices (e.g., mobile phonesor mobile phone applications). A user interface device 440 may also actas a sensing device and relay data to a gateway device 450 or directlyto remote servers 460 through the communication network 480.

FIG. 5 illustrates an example of a user interface and sensing device.Specifically, FIG. 5 illustrates an on-body sensor 510. The on-bodysensor 510 may be a fall and movement sensor with an emergency button.The on-body sensor 510 is intended to be worn and easily attached tomany articles of clothing on the trunk (e.g., belt, lapel, brazier,lanyard, etc.).

FIG. 6 illustrates a device 600 that represents an example of theon-body sensor 510. In order to facilitate wearability, the device 600embodies a clip form factor. The clip is fastened closed through tensionwhen no force is applied, but can be opened upon demand (e.g., similarto a clothes pin), thereby ensuring that it remains connected to anarticle of clothing.

When no force is applied to the clip, both sides of the device 600 arein contact with one another. The device 600 includes compliance contacts(e.g., an electrical switch) comprising a conductive contact on eachside of the clip. When the clip is forced open or clipped around a pieceof fabric, the switch is opened. Otherwise, the switch is closed and thecircuit loop completed. Using the compliance contacts, the system 400can identify whether the sensor is being worn. This information can beused to identify false falls created from dropping or otherwise handlingthe device 600 when not worn. The user can also be reminded via audibleor visual interfaces based on the system 400 detecting that the device600 is not being worn as a result of the output of the compliancecontacts. In determining whether to provide the reminder, the system 400may consider other sensors within the monitored premise. For instance,the system 400 may detect motion within the monitored premise based onoutput of one or more motion sensors, determine that the device 600 isnot being worn based on output from the compliance contacts, and providea reminder to wear the device 600 based on the determination that thedevice 600 is not being worn at a time when motion has been detected inthe monitored premise.

Referring again to FIG. 5, the on-body sensor 510 comprises multi-modalsensing (e.g., triaxial inertial sensor, angular rate sensor,magnetometer, barometric pressure sensor, etc.), input/output, radio(e.g., via input/output), a processor, memory, battery, and userinterface capabilities for human interaction (e.g., a button, LED/LCD,buzzer, etc.). The on-body sensor 510 may be used to measure gross humanmotion and activity, detect specific events or behaviors (e.g., falls,walking, running, sleeping, etc.), communicate to the user (e.g.,reminders, notifications, etc.), or capture user input (e.g., panicbutton press, verification of event, etc.). Detecting falls with on-bodysensing is described in further detail below.

Referring again to FIG. 4, a gateway device 450 can be used to relayinformation between remote servers 460 (e.g., over public or privatecommunication network) and systems at the user location. The gatewaydevice 450 can also allow systems within a user's location tocommunicate without involvement from remote servers 460. Certainincarnations of the system 400 may not include a gateway device 450.Therefore, passive sensors 410, assistance devices 420, imaging sensors430, and/or user interface devices 440 may be connected directly to thecommunication network 480.

Remote servers 460 may be employed to store, process, and initiateactions based upon fall, fall-related, or other data collected abouteach monitored user and location. Monitoring center agents can alsoannotate user records stored on the remote servers 460.

A monitoring center 470 may employ automated or human agents to observeusers' fall-related events and contact users or caregivers based ondefined protocols, quantitative or qualitative assessments. Monitoringcenter agents can also annotate user records stored on the remote server460.

FIG. 7 illustrates an example process 700 for fall management. Theoperations of the example process 700 are described generally as beingperformed by the system 400. The operations of the example process 700may be performed by one of the components of the system 400 or may beperformed by any combination of the components of the system 400. Theoperations of the example process 700 also may be performed by one ofthe components of the system 200 or may be performed by any combinationof the components of the system 200. In some implementations, operationsof the example process 700 may be performed by one or more processorsincluded in one or more electronic devices.

The fall management process 700 includes data capture (710), falldetection (720), fall verification (730), fall risk assessment (740),fall risk reduction (750), and reporting (760). Although several stepsare illustrated as part of the fall management process 700, some fallmanagement implementations may only employ a subset of these steps.

The system 400 performs data capture (710). Data can be captured fromone or more passive sensors, imaging sensors, assistance devices, anduser interface devices. Data can be unprocessed sensor readings orsensor-processed readings. Data capture can be triggered by the passivesensors, imaging sensors, user interface devices, remote servers, ormonitoring center. Data capture can consist of instantaneous orcontinuously sampled readings. Data can be forwarded directly fromdevices to remote servers or to remote servers via a gateway device.Remote servers, a gateway device, or sensors may coordinate the captureof data or buffer data to facilitate on-sensor, on-gateway, or remoteprocessing. In addition to raw sensor readings, meta-data encompassingsensor location, timestamp, etc. can be forwarded to other devices,sensors, gateways, or remote servers.

The system 400 performs fall detection (720). Falls can be detectedindependently by passive sensors, imaging sensors, or user interfacedevices (e.g., on-body sensor). Each device can classify a possible falland communicate fall events or quantitative metrics related to thepossibility of a fall (e.g., fall classification score). For example, anon-body sensor can capture human motion and detect motioncharacteristics indicative of a fall (described in more detail below).Furthermore, an image sensor can detect the likelihood of a fall throughanalysis of images and other in-device sensors (described in more detailbelow).

Fall detection may also be accomplished through the use of multiplesensors in parallel (e.g., hierarchical) or sequentially to improvesensitivity and specificity of fall detection. Numerous examples ofcombined sequential and parallel fall detection may be used and datafrom any combination of the sensors described throughout this disclosuremay be fused and considered in combination to detect a potential fallevent. For example, the system 400 may detect entry into a room based onoutput from a motion sensor and/or a door sensor. In this example, thesystem 400 detects that the room has not been exited after a thresholdperiod of time has passed since the room entry was detected and detectssensor inactivity across all sensors after the room entry was detected.Based on the detections made and consideration of output of all of thesensors within the system 400, the system 400 determines that apotential fall event may have occurred in the room and, in response tothe determination that a potential fall event may have occurred in theroom, initiates further processing to verify whether a potential fallevent has occurred in the room.

In another example, the system 400 detects a potential fall event basedon output from an on-body sensor. In this example, the system 400controls an imaging sensor to capture one or more images in a room wherethe potential fall event is expected to have occurred, performs analysisof the captured images, and detects possible presence of a proneindividual on the ground in the room. The system 400 also detects sensorinactivity across all sensors after detecting the potential fall eventbased on output from the on-body sensor. Based on the detections madeand consideration of output of all of the sensors within the system 400,the system 400 determines that a potential fall event may have occurredin the room and, in response to the determination that a potential fallevent may have occurred in the room, initiates further processing toverify whether a potential fall event has occurred in the room.

Independent fall detection processes on single devices or groups ofdevices also may be weighted (e.g., based on confidence or accuracy offall detection efficacy). Such weighting may be used to compute anaggregate score indicative of the confidence of a possible fall. Weightsmay be assigned based on currently observed data and conditions,historic data from the monitored individual, or population data. Falldetection sensitivity may be configured by the user based onmanipulation of weights associated with any of the aforementioned steps.For example, fall sensitivity could be set by adjusting the interval ofsensed inactivity or the threshold for decreased activity. The system400 may consider output from any of the sensors in the system 400 incomputing the aggregate score. The system 400 may use the aggregatescore to detect a potential fall event by comparing the aggregate scoreto a threshold. For instance, the system 400 detects a potential fallevent based on the comparison of the aggregate score to the thresholdrevealing that the aggregate score meets the threshold and determinesthat a potential fall event has not occurred based on the comparison ofthe aggregate score to the threshold revealing that the aggregate scoredoes not meet the threshold. By considering weighted output from manydifferent sensors and fall detection processes in computing theaggregate score, the system 400 may provide more accurate fall detectionwith a lower false positive rate because detection of a fall detectiononly occurs when several sensors sense potential fall criteria or asingle sensor detects a very high likelihood of a potential fall.

The system 400 performs fall verification (730). If a likely fall isdetected, the detecting device, gateway, remote server, or monitoringcenter can initiate fall verification. The process can include anautomated or human-prompted user response. For example, a user may bealerted (e.g., by audible tone, vibration, human operator, automatedoperator, or visual indicator) to verify their need for help (e.g., abutton press or vocal response) or may be alerted to respond within aperiod of time to cancel a potential fall event. A human operator mayalso speak and listen to a user over two-way communication link.

Fall verification also may be made by human inspection of capturedimages. For example, following the detection of a potential fall event,an image or successive images captured proximal to the fall may be sentto the monitoring center for human verification. Image capture also maybe triggered post fall (e.g., by a monitoring center or by othercaregivers) to verify a fall event. Other contextual sensor or meta-datamay be forwarded to human responders to assist in the verification offall.

Fall verification procedures may be staged sequentially or paired withfall detection mechanisms to create a hierarchical fall escalationprocess. For example, less accurate fall detection methods may triggerless invasive user verification (e.g., prompted user button press). Ifno user response is given within a threshold period of time, then moreaccurate fall detection methods may be employed alongside more invasivefall verification (e.g., two way communications with monitoring center).

The system 400 performs fall risk assessment (740). Assessment of fallrisk may be made on the basis of data captured by sensors, userinterface devices, or historic and stored data. Measures such as gaitspeed and balance can be directly assessed via passive and userinterface devices. For example, two motion sensors placed in a hallwaycan measure gait speed and balance can be assessed via an on-body userinterface device (e.g., via on-board inertial sensor and angular ratesensor). Other behavioral data such as medication adherence, sleeppatterns, kitchen or restroom use can be used to augment mobilitymetrics. Data can be combined with prior knowledge of fall incidents orpreviously verified fall events. In addition, users may be prompted tosubmit responses to questions or requests for information (e.g., via auser interface device or website, electronic medical records, residencelayout, etc.) to form an aggregate fall risk assessment score. Scorescan be computed, compared, or modified against individual or populationscores and histories. Scores can also be computed for various timescalesand locations. Fall risk assessment may also take into considerationtrending of scores for an individual.

The system 400 performs fall risk reduction (750). Various assistiveapproaches may be employed with or without prior fall risk assessmentscoring to help reduce fall risk. Assistance devices such as automatedlighting or medication dispensers can be used to reduce environmentalhazards or behaviors related to increase in fall risk, respectively.Assistance devices may be triggered by fall assessment scores, othersensors, user interface devices, or remote servers. For example,automated lighting can be turned-on when a user gets out of bed.

Furthermore, notifications or educational material can be delivered(e.g., by default, for certain fall risk assessment scores, for certainevents, etc.) to the user (e.g., via a user interface device or otheroutput device) to help the user better understand and correct fall riskfactors. Tips or behavior change techniques can help the user set up asafer environment or promote behaviors associated with decreased fallrisk. Notifications may be combined with sensing or other user interfaceprompts (e.g., prompts to answer questionnaires) to assess adherence tofall risk reduction techniques in real-time or across a period of time.Users may be scored on their ability to reduce fall risk at varioustimescales or in various locations. Fall risk reduction scores may becompared to individual or population historic data.

The system 400 performs reporting (760). Fall risk, detection, andprevention data, scores, annotations, or observations can be stored atthe remote server. Data can be compiled and reported to users,caregivers, monitoring centers, or other trusted parties. Data(including timestamps, scores, locations, confidence, etc.) can be usedfor the purposes of response to events, for preventative fall riskreduction strategies, or by professional caregivers for general healthassessment. Data or scores can be compared to individual or populationdata and reported to all aforementioned parties when appropriate. Datareporting may be combined with prompts for data entry. For example, auser could receive a notification that bathroom habits are abnormal andbe asked whether they are feeling well. Access to reported data can berestricted based on preferences of the user or caregivers.Notifications, reminders, user prompts, questionnaires, monitoredresponses, and other user interface modes can be configured by ruleswith associated parameters. Rules can be stored and executed at theremote server, gateway device, sensors, or user interface devices.

FIG. 8 illustrates an example process 800 for fall detection using anon-body user interface device. The operations of the example process 800are described generally as being performed by the system 400. Theoperations of the example process 800 may be performed by one of thecomponents of the system 400 or may be performed by any combination ofthe components of the system 400. The operations of the example process800 also may be performed by one of the components of the system 200 ormay be performed by any combination of the components of the system 200.In some implementations, operations of the example process 800 may beperformed by one or more processors included in one or more electronicdevices.

In order to accurately detect a fall event, the on-body user interfacedevice identifies the various characteristics of a fall comprised of theuser starting from a standing or sitting position, falling through tothe ground, impacting a surface, and remaining inactive after the fall.The user's trunk may transition from a vertical to horizontal position.This may result in a ninety degree change in trunk orientation, butsince the user may not be standing straight before the fall, or may notbe prone or supine after the fall, the angle may not reach ninetydegrees. FIG. 8 illustrates a fall detection process 800 for userswearing an on-body sensor with continuous sensing and detection.

The system 400 triggers fall detection processing based on detection ofa fall-related signature (810). The fall detection process may betriggered by an impact metric (e.g., measured from inertial sensing) ora similar fall-related signature (e.g., free fall) crossing a minimumthreshold. The fall-related signature may be quantified and stratifiedinto defined ranges indicative of fall detection confidence.

For instance, FIG. 9 illustrates example fall detection criteria. Thefall detection criteria include a range of impact metrics 910 used toquantify a measured impact metric. As shown, the range of impact metricsmay include less than two, between two to five, between five to ten,between ten to fifteen, and greater than fifteen. The system 400 may usethe impact metric of two as a threshold for triggering fall detectionprocessing. For instance, the system 400 quantifies a measured impactwithin the ranges of impact metrics and determines not to trigger falldetection processing based on the measured impact falling within therange of less than two. For any of the other ranges, the system 400triggers fall detection processing and records the range in which themeasured impact falls for later processing.

Referring again to FIG. 8, the system 400 calculates orientation changebased on triggering fall detection processing (820). Based on the system400 detecting that a measured impact or similar metric crosses thepreviously mentioned minimum threshold, the system 400 calculates anorientation change using inertial or angular rate measures from beforeand after the detected impact or other event. The orientation value maybe quantified and stratified into defined ranges.

For example, the fall detection criteria shown in FIG. 9 include a rangeof orientation changes 920 used to quantify an orientation change. Asshown, the range of orientation changes may include less than fifty,between fifty to sixty, between sixty to seventy-five, betweenseventy-five to eighty-five, and greater than eighty-five. The system400 may use the orientation change of fifty as a threshold forcontinuing fall detection processing. For instance, the system 400quantifies a calculated orientation change within the ranges oforientation changes and determines not to continue fall detectionprocessing based on the calculated orientation change falling within therange of less than fifty. For any of the other ranges, the system 400continues fall detection processing and records the range in which thecalculated orientation change falls for later processing.

Referring again to FIG. 8, the system 400 determines a minimum requiredinactivity period based on the fall-related signature and theorientation change (830). Based on the defined ranges derived fromimpact/signature scoring and orientation scoring, a minimum requiredinactivity period can be determined by a lookup table or functionalrelationship.

The fall detection criteria shown in FIG. 9 include an inactivity periodlookup table 930. The system 400 references the lookup table 930 usingthe range of the measured impact and the range of the calculatedorientation change and sets the minimum required inactivity period asthe period of time defined by the appropriate entry in the lookup table930. For example, with an impact metric greater than ten, but less thanfifteen, and an orientation change greater than eighty-five, theinactivity period is set as low as thirty seconds to signal a likelyfall.

Referring again to FIG. 8, the system 400 detects a potential fall eventbased on monitoring activity during the minimum required inactivityperiod (840). The system 400 may monitor output of the on-body sensorand output from any of the other sensors in the system 400 and determinewhether any of the sensors signal activity. The system 400 continues tomonitor the sensor output until the set period of inactivity has beenreached and the system 400 detects a potential fall event based ondetermining that the set period of inactivity has passed withoutdetecting sensed activity from any of the sensors in the system 400.

FIG. 10 illustrates an example process 1000 for tuning sensitivity andspecificity of fall detection. The operations of the example process1000 are described generally as being performed by the system 400. Theoperations of the example process 1000 may be performed by one of thecomponents of the system 400 or may be performed by any combination ofthe components of the system 400. The operations of the example process1000 also may be performed by one of the components of the system 200 ormay be performed by any combination of the components of the system 200.In some implementations, operations of the example process 1000 may beperformed by one or more processors included in one or more electronicdevices.

To tune sensitivity and specificity of fall detection (e.g., the on-bodyfall detection process 800), the process 1000 uses user feedback. Theprocess 1000 may produce more granular fall reporting (e.g., true,false, minor, canceled falls) and may help to reduce and reportincidence of false positives or false negatives.

The system 400 detects a potential fall event (1005). A possible fall isdetected by the on-body device, by other sensors, or user interfacedevices. Any of the techniques described throughout this disclosure maybe used to detect a potential fall event.

The system 400 prompts the user for cancellation of the potential fallevent (1010). A user prompt may be initiated (e.g., audible or visual).The user can respond (e.g., by a button press or vocalization) to theuser prompt at the device to cancel the detected potential fall event.

The system 400 determines whether the user cancels the potential fallevent within a defined period of time (1015). For instance, the system400 monitors for input cancelling the potential fall event until thedefined period of time has been reached and the system 400 determineswhether the user cancelled the potential fall event within the definedperiod of time based on the monitoring. Based on a determination thatthe potential fall event was not cancelled within the defined period oftime, the system 400 generates a fall signal (e.g., a fall signal fromthe body-worn device).

Based on a determination that the potential fall event was cancelledwithin the defined period of time, the system 400 makes a measurement ofoverall activity over the minimum inactivity period previously mentioned(1020). For example, the system 400 measures the activity detected bythe on-body sensor after detection of the potential fall event until theinput cancelling the potential fall event was received.

The system 400 determines whether the measurement of overall activitymeets an expected maximum activity (1025). For instance, the system 400compares the measurement of overall activity to the expected maximumactivity and determines whether the measurement of overall activitymeets the expected maximum activity based on the comparison.

Based on a determination that the measurement of overall activity meetsthe expected maximum activity, the system 400 signals a false falldetection (1030). For example, the system 400 classifies the sensor dataused to detect the potential fall event as being sensor data associatedwith a false detection of a potential fall event. In this example, thesystem 400 may tune the potential fall detection process such thatsensor data similar to the sensor data associated with the falsedetection of the potential fall event does not result in detection of apotential fall event in the future.

Based on a determination that the measurement of overall activity doesnot meet the expected maximum activity, the system 400 measures postureor orientation (1035) and determines whether the subject recovered fromthe suspected fall based on the measured posture or orientation (1040).For instance, the system 400 analyzes the measured posture ororientation and determines whether the subject has returned to anupright position.

Based on a determination that the subject recovered from the suspectedfall, the system 400 triggers a minor fall (1045). For example, thesystem 400 classifies the sensor data used to detect the potential fallevent as being sensor data associated with a minor fall. In thisexample, the system 400 may tune the potential fall detection processsuch that sensor data similar to the sensor data associated with theminor fall results in detection of a minor fall event in the future. Thesystem 400 may handle minor fall events differently than regular fallevents. For instance, the system 400 may wait longer to see if a patientrecovers from a minor fall prior to alerting a remote caregiver ormonitoring station.

Based on a determination that the subject did not recover from thesuspected fall, the system 400 performs another user prompt forcancellation (1050) and determines whether the user cancels thepotential fall event within a defined period of time from the additionalprompt for cancellation (1055). Based on a determination that thepotential fall event was cancelled within the defined period of time,the system 400 signals a cancelled fall (1060). For instance, the system400 does not provide an alert for the potential fall event, but doesclassify the sensor data used to detect the potential fall event asbeing sensor data associated with a fall that was ultimately cancelled.

Based on a determination that the potential fall event was not cancelledwithin the defined period of time, the system 400 generates a fallsignal (1065). For instance, the system 400 may generate a fall signalfrom the body-worn device. The fall signal may be sent to a remotecaregiver or monitoring station to alert the remote caregiver ormonitoring station to provide assistance to the patient who experiencedthe potential fall event.

Granular fall detection classes such as true fall, false fall, minorfall, and cancelled fall can be used to tune system parameters for eachindividual user, provide caregivers or trusted individuals with falldata, and provide automated mechanisms for fall verification.Furthermore, the data can be stored at the remote servers.

FIG. 11 illustrates an example process 1100 for fall detection andreporting. The operations of the example process 1100 are describedgenerally as being performed by the system 400. The operations of theexample process 1100 may be performed by one of the components of thesystem 400 or may be performed by any combination of the components ofthe system 400. The operations of the example process 1100 also may beperformed by one of the components of the system 200 or may be performedby any combination of the components of the system 200. In someimplementations, operations of the example process 1100 may be performedby one or more processors included in one or more electronic devices.

In general, the process 1100 enables fall detection and reporting basedon human movement analysis. The system 400 performs a triggered orscheduled image capture (1110). For example, the system 400 may triggera camera on an image sensing device to capture an image based on eventsdetected by one or more of the image sensing device's sensors (e.g.,perceived motion passive infrared motion sensor, triaxial inertialsensor). In this example, movement or impact detected proximal to theimage sensing device may initiate the capture of an image. Furthermore,the system 400 may trigger the camera by one or more external sensorsinterfaced via a gateway device. For instance, the press of a panicbutton or the opening of a door sensor may trigger one or more imagesensing devices to capture an image. Finally, image capture may bescheduled (e.g., capture an image every one minute during the hours ofsix in the morning through ten in the evening). In lower lightconditions (e.g., characterized by the illumination sensor), the system400 may employ infrared illumination to increase image detail andquality.

After image capture, the system 400 performs image foregroundsegmentation and filtering (1120). The system 400 (e.g., the imagesensing device) may perform image foreground segmentation via backgroundsubtraction or other averaging approaches. The system 400 may filtercaptured images to help reduce foreground noise and isolate largeregions of change. The process may identify changed pixels from previousimages, including those morphologically likely to represent human formsor shapes.

After image foreground segmentation and filtering, the system 400performs human segmentation (1130). The system 400 segments possiblehuman shapes via template matches, shape fitting, or similar methods.For example, the system 400 may segment a foreground shape fallingwithin an approximate elliptical boundary over a size threshold. Suchsegmentation may reduce incidence of false detection and reporting(e.g., small pet activity). To further reduce incidence of falsedetection and reporting, the system 400 may remove regions of thecamera's field of view from analysis. For instance, if a bed werepresent in the field of view, the bed may be marked as a non-detectionregion and the system 400 would not analyze that portion of imagescaptured by the image sensing device.

After human segmentation, the system 400 performs human orientation andposition estimation (1140). For example, the system 400 calculatesorientation (e.g., human shape upright, angled, prone, etc.) andposition (e.g., human shape above floor, near floor, etc.) by templateor boundary shape proportion and rotation relative to a horizontal imageplane. This estimation enables identification of postures and restingpositions indicative of a fall. The floor proximal planar boundary canbe specifically defined and moved to fit the unique geometries ofdifferent rooms.

After human orientation and position estimation, the system 400 performssuccessive image and/or sensor data comparison (1150). For example, thesystem 400 stores, either on or off the image sensing device, theorientation and position information calculated previously and comparesthe prior orientation and position information with successive imageorientations and positions. The system 200 repeats this process andisolates changes in position and orientation indicative of a fall (e.g.,movement towards the ground), or relative stasis of position andorientation indicative of a fall (e.g., incapacitation after a fall).Furthermore, the system 400 combines motion sensor information with orused independent of image-based analysis to ascertain movement throughhorizontal planes of motion (e.g., human falling from an uprightposition).

The system 200 performs inactivity detection (1160). For example, thesystem 400 detects periods of relative inactivity, such as thosefollowing a potential fall, from lack of or decreased motion, inertialmeasures, image-derived orientation and position information, externalsensors, or even a combination thereof. The system 400 may classifylonger periods of relative inactivity as being indicative of a fall, andclassify shorter periods of relative inactivity as being indicative of anon-fall event or recovery from a fall.

After inactivity detection, the system 400 performs fall classification(1170). The system 400 may combine (e.g., logically or algebraically)the data and information compiled in previous operations of the process1100 and use the combined data in several ways to classify possiblefalls. For example, if an impact is detected, orientation and positionare indicative of a human in a fallen state, and a period of inactivityhas exceeded a defined threshold, then the system 400 classifies theevent as a fall. Classification sensitivity may be configured by theuser based on manipulation of variables associated with any of theaforementioned steps. For example, fall sensitivity could be set byadjusting the interval of sensed inactivity or the threshold fordecreased activity. Not all prior conditions must be met, nor all priorsteps completed, for fall classification. The system 400 may reportclassification confidence based on the quality of inputs or classifierperformance. Furthermore, the system 400 may implement the classifier ina variety of ways such as, but not limited to an expert system, nativeBayes, decision tree, neural network, etc.

After fall classification, the system 400 performs fall reporting(1180). For example, potential fall events are forwarded to a gatewaydevice, remote monitoring servers, and ultimately to users or centralmonitoring station(s) if appropriate rules and preferences are met.Images (e.g., past and present), data, and location information can besent for purposes of reporting and verification. Moreover, potentialnon-fall events, images, data, and location can be forwarded to users orcentral monitoring station(s) for verification. Verification of fallevents is not a requisite function of the system, rather an additionalfeature. Fall detection can be performed with or without image or otherhuman-based verification.

FIG. 12 shows fall detection examples with three possible scenarios thatillustrate aspects of the fall detection process 1000 discussed above.In the first scenario (a), a person stands upright in a room. In thesecond scenario (b), a person has fallen and is prone on the floor. Inthe third scenario (c), a person has fallen to a slumped position on thefloor. Illustrations (d), (e), and (f) represent the results offoreground separation and filtering of illustrations (a), (b), and (c),respectively. Illustrations (g), (h), and (i) represent the results ofhuman orientation and position estimation and inactivity detection (asdenoted by a clock) of the previous illustrations, respectively. Noticein illustration (g) that the human shape estimator, illustrated as anellipse, but not limited to ellipses, extends beyond a floor proximalplanar boundary; whereas in illustrations (h) and (i), the human shapeestimators are below the plane and their orientations are not vertical,hence, inactivity detection has commenced.

Analysis of room geometry within captured images may be used to projecta virtual plane of where a person should be oriented below in a fallevent. The system 400 may analyze floor geometry and then performcentroid-based processing to determine where the floor is located in thecaptured images. After determining the location of the floor, the system400 projects the virtual plane within the captured images at aparticular distance above the floor.

In some implementations, the image sensing device and optional triggersources (e.g., other sensors) communicate to a gateway (e.g., homemonitoring panel) within a home or facility. The gateway's memoryenables buffering of images and data from the image sensor and othersensors. Data is forwarded to remote monitoring servers over a longrange wireless network (e.g., cellular link). Rules and preferences setat the remote monitoring server enable potential fall information (e.g.,captured images and data) to be forwarded via an IP network to users ora central monitoring station for fall verification. If a fall isverified by human inspection of captured images and data, a response canbe initiated (e.g., a two-way voice call may be established with thegateway device, emergency responders may be dispatched, etc.) andlocation information from the system can be communicated to thoseproviding assistance.

In some implementations, the system (e.g., the system 200 or the system400) may evaluate context in determining how to handle a fall detectionevent. In these implementations, the system may check other activity inthe property and determine how to handle the fall detection event basedon the other activity. For instance, when the system detects otheractivity in the property, the system may attempt to alert someone in theproperty to the potential fall event (e.g., by providing an audiblealert in the home that indicates the fall detection event). When thesystem does not detect other activity in the property, the system may,based on the fall detection event, send electronic messages to acaregiver associated with the property to alert the caregiver to thefall detection event, establish a two-way voice communication sessionwith a monitoring system at the property, and/or dispatch emergencyservices.

In some examples, the system may tune sensitivity of one or moresensors/contexts used in fall detection and may determine a score aspart of fall classification. In these examples, the system may determinethe score based on a number of sensors that indicate a potential fall.For instance, the system may determine a relatively high score when thesystem detects a thud based on an accelerometer sensor, detects multiplemotion sensors indicating motion consistent with a fall, and performsimage analysis that suggests that a person has moved from a verticalorientation to a horizontal orientation below a plane near the floor.The system may determine a relatively low score when the system onlyperforms image analysis that suggests that a person is horizontallyoriented below a plane near the floor. The system may consider thenumber of motion sensors detecting motion and leverage all sensor data.The system may typically operate using a subset of sensors and move to aprocess that leverages all sensors when a potential fall is detected bythe subset of sensors. The system may consider historic data (e.g.,classification by caregivers of whether a fall detection event wasactually a fall or a mistake) and tune fall detection based on thehistoric data.

In some implementations, the location in the home where the falloccurred may be determined and communicated to an emergency responseteam. In addition, the location in the home where the fall occurred maybe used to pick the other sensors the system reviews in confirming apotential fall event. For instance, when the system determines that thepotential fall occurs in the basement, the system determines not toconsider sensors in the upstairs bedroom, as the sensors in the upstairsbedroom are unlikely to be relevant to the potential fall event in thebasement.

The described systems, methods, and techniques may be implemented indigital electronic circuitry, computer hardware, firmware, software, orin combinations of these elements. Apparatus implementing thesetechniques may include appropriate input and output devices, a computerprocessor, and a computer program product tangibly embodied in amachine-readable storage device for execution by a programmableprocessor. A process implementing these techniques may be performed by aprogrammable processor executing a program of instructions to performdesired functions by operating on input data and generating appropriateoutput. The techniques may be implemented in one or more computerprograms that are executable on a programmable system including at leastone programmable processor coupled to receive data and instructionsfrom, and to transmit data and instructions to, a data storage system,at least one input device, and at least one output device. Each computerprogram may be implemented in a high-level procedural or object-orientedprogramming language, or in assembly or machine language if desired; andin any case, the language may be a compiled or interpreted language.Suitable processors include, by way of example, both general and specialpurpose microprocessors. Generally, a processor will receiveinstructions and data from a read-only memory and/or a random accessmemory. Storage devices suitable for tangibly embodying computer programinstructions and data include all forms of non-volatile memory,including by way of example semiconductor memory devices, such asErasable Programmable Read-Only Memory (EPROM), Electrically ErasableProgrammable Read-Only Memory (EEPROM), and flash memory devices;magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and Compact Disc Read-Only Memory (CD-ROM). Anyof the foregoing may be supplemented by, or incorporated in,specially-designed ASICs (application-specific integrated circuits).

It will be understood that various modifications may be made. Forexample, other useful implementations could be achieved if steps of thedisclosed techniques were performed in a different order and/or ifcomponents in the disclosed systems were combined in a different mannerand/or replaced or supplemented by other components. Accordingly, otherimplementations are within the scope of the disclosure.

What is claimed is:
 1. A method comprising: monitoring output from atleast one sensor configured to sense, in a room of a building, activityassociated with a patient falling; based on the monitoring of outputfrom the at least one sensor, determining to capture one or more imagesof the room; based on the determination to capture one or more images ofthe room, capturing, with a camera positioned to include the patientwithin a field of view of the camera, an image of the room; analyzingthe captured image of the room to detect a state of the patient at atime of capturing the image; determining, based on the detected state ofthe patient, a potential fall event for the patient; and based on thedetermination of the potential fall event for the patient, sending, by acommunication device, a message indicating the potential fall event forthe patient, wherein the at least one sensor configured to senseactivity associated with the patient falling is an on-body sensorconfigured to detect an impact and determine a change in an orientationof the patient; wherein monitoring output from the at least one sensorcomprises: receiving data indicating a detected change in an orientationof the patient and an amount of the orientation change, receiving dataindicating a detected impact and a severity of the detected impact,determining, based on the received amount of orientation change and thereceived data indicating the severity of the impact of the patient, athreshold for inactivity of the patient, and determining, based onoutput from the on-body sensor, that the patient has been inactive for aperiod of time greater than the determined threshold; and whereindetermining to capture one or more images of the room comprisesdetermining to capture one or more images of the room based on thedetermination that the patient has been inactive for a period of timegreater than the determined threshold.
 2. A method comprising:monitoring output from at least one sensor configured to sense, in aroom of a building, activity associated with a patient falling; based onthe monitoring of output from the at least one sensor, determining tocapture one or more images of the room; based on the determination tocapture one or more images of the room, capturing, with a camerapositioned to include the patient within a field of view of the camera,an image of the room; analyzing the captured image of the room to detecta state of the patient at a time of capturing the image; determining,based on the detected state of the patient, a potential fall event forthe patient; and based on the determination of the potential fall eventfor the patient, sending, by a communication device, a messageindicating the potential fall event for the patient, wherein the atleast one sensor configured to sense activity associated with thepatient falling is a sensor configured to determine a presence of thepatient in the room; and wherein monitoring output from the at least onesensor comprises: receiving, from the sensor configured to determine thepresence of the patient in the room, a signal indicating that thepatient is present in the room, and after a threshold period of time,determining that the patient has not left the room and that no furthersignals have been received from the sensor configured to determine thepresence of the patient in the room; and wherein determining to captureone or more images of the room comprises determining to capture one ormore images of the room based on determining that the patient has notleft the room and that no further signals have been received from thesensor configured to determine the presence of the patient in the room.3. A method comprising: monitoring output from at least one sensorconfigured to sense, in a room of a building, activity associated with apatient falling; based on the monitoring of output from the at least onesensor, determining to capture one or more images of the room; based onthe determination to capture one or more images of the room, capturing,with a camera positioned to include the patient within a field of viewof the camera, an image of the room; analyzing the captured image of theroom to detect a state of the patient at a time of capturing the image;determining, based on the detected state of the patient, a potentialfall event for the patient; and based on the determination of thepotential fall event for the patient, sending, by a communicationdevice, a message indicating the potential fall event for the patient,wherein analyzing the captured image of the room to detect a state ofthe patient at a time of capturing the image comprises: performing imageforeground segmentation on the captured image to create a segmentedimage, performing template matching on the segmented image to identify ahuman shape in the segmented image, and calculating a position and anorientation associated with the identified human shape in the segmentedimage, and wherein determining the potential fall event for the patientcomprises determining a potential fall event for the patient based onthe calculated position and the calculated orientation.
 4. The method ofclaim 3, wherein determining the potential fall event for the patientbased on the calculated position and the calculated orientationcomprises: monitoring successive image and sensor data after calculatingthe position and the orientation; comparing the successive image andsensor data with prior image and sensor data; determining an activitylevel of the patient based on the comparison of the successive image andsensor data with the prior image and sensor data; classifying thepotential fall event based on the determined activity level of thepatient; and handling reporting for the potential fall event based onthe classification of the potential fall event.
 5. A method comprising:monitoring output from at least one sensor configured to sense, in aroom of a building, activity associated with a patient falling; based onthe monitoring of output from the at least one sensor, determining tocapture one or more images of the room; based on the determination tocapture one or more images of the room, capturing, with a camerapositioned to include the patient within a field of view of the camera,an image of the room; analyzing the captured image of the room to detecta state of the patient at a time of capturing the image; determining,based on the detected state of the patient, a potential fall event forthe patient; based on the determination of the potential fall event forthe patient, sending, by a communication device, a message indicatingthe potential fall event for the patient; analyzing the monitored outputfrom the at least one sensor over a period of time to determineactivities of the patient over the period of time; accessing informationindicative of expected activities of the patient over the period oftime; comparing the determined activities of the patient over the periodof time to the expected activities of the patient over the period oftime; and based on the comparison revealing that the determinedactivities of the patient over the period of time do not match theexpected activities of the patient over the period of time, determininga level of fall risk associated with the patient.
 6. The method of claim5, further comprising: determining that the level of fall riskassociated with the patient exceeds a threshold; and based on thedetermination that the level of fall risk associated with the patientexceeds the threshold, sending a message to a monitoring server that islocated remotely from the building.
 7. The method of claim 5, furthercomprising: determining that the level of fall risk associated with thepatient exceeds a threshold; and based on the determination that thelevel of fall risk associated with the patient exceeds the threshold,automatically performing one or more operations to reduce the level offall risk associated with the patient.
 8. A method comprising:monitoring output from at least one sensor configured to sense, in aroom of a building, activity associated with a patient falling; based onthe monitoring of output from the at least one sensor, determining tocapture one or more images of the room; based on the determination tocapture one or more images of the room, capturing, with a camerapositioned to include the patient within a field of view of the camera,an image of the room; analyzing the captured image of the room to detecta state of the patient at a time of capturing the image; determining,based on the detected state of the patient, a potential fall event forthe patient; and based on the determination of the potential fall eventfor the patient, sending, by a communication device, a messageindicating the potential fall event for the patient, wherein sending themessage indicating the potential fall event for the patient comprisessending, to the patient, the message indicating the potential fall eventand providing the patient with an opportunity to cancel the potentialfall event.
 9. The method of claim 8, further comprising: determiningthat the patient has not cancelled the potential fall event within athreshold period of time; and based on determining that the patient hasnot cancelled the potential fall event within the threshold period oftime, sending a message to a monitoring server indicating the potentialfall event.
 10. The method of claim 8, further comprising: receiving,from the patient, an indication to cancel the potential fall event; andbased on receiving the indication to cancel the potential fall event,determining an overall activity of the patient between detecting thepotential fall event and receiving the indication to cancel thepotential fall event.
 11. The method of claim 10, further comprising:determining that the overall activity of the patient is above athreshold of activity; and based on determining that the overallactivity of the patient is above the threshold of activity, signalingthat the potential fall event was detection of a false fall.
 12. Themethod of claim 10, further comprising: determining that the overallactivity of the patient is below a threshold of activity; and based ondetermining that the overall activity of the patient is below thethreshold of activity, determining an orientation of the patient. 13.The method of claim 12, further comprising: determining that thedetermined orientation of the patient is upright; and based ondetermining that the determined orientation of the patient is upright,signaling that the potential fall event was detection of a minor fall.14. The method of claim 12, further comprising: determining that thedetermined orientation of the patient is not upright; and based ondetermining that the determined orientation of the patient is notupright, sending another message to the patient that provides thepatient with another opportunity to cancel the potential fall event. 15.The method of claim 14, further comprising: determining that the patienthas not cancelled the potential fall event within a threshold period oftime after sending another message to the patient that provides thepatient with another opportunity to cancel the potential fall event; andbased on determining that the patient has not cancelled the potentialfall event within the threshold period of time after sending anothermessage to the patient that provides the patient with anotheropportunity to cancel the potential fall event, sending a message to amonitoring server indicating the potential fall event.
 16. The method ofclaim 14, further comprising: after sending another message to thepatient that provides the patient with another opportunity to cancel thepotential fall event, receiving, from the patient, an indication tocancel the potential fall event; and based on receiving the indicationto cancel the potential fall event, signaling that the potential fallevent was a cancelled fall event.
 17. A system comprising: at least oneprocessor; and at least one memory coupled to the at least one processorhaving stored thereon instructions which, when executed by the at leastone processor, causes the at least one processor to perform operationscomprising: monitoring output from at least one sensor configured tosense, in a room of a building, activity associated with a patientfalling; based on the monitoring of output from the at least one sensor,determining to capture one or more images of the room; based on thedetermination to capture one or more images of the room, capturing, witha camera positioned to include the patient within a field of view of thecamera, an image of the room; analyzing the captured image of the roomto detect a state of the patient at a time of capturing the image;determining, based on the detected state of the patient, a potentialfall event for the patient; and based on the determination of thepotential fall event for the patient, sending, by a communicationdevice, a message indicating the potential fall event for the patient;wherein the at least one sensor configured to sense activity associatedwith the patient falling is an on-body sensor configured to detect animpact and determine a change in an orientation of the patient; whereinmonitoring output from the at least one sensor comprises: receiving dataindicating a detected change in an orientation of the patient and anamount of the orientation change, receiving data indicating a detectedimpact and a severity of the detected impact, determining, based on thereceived amount of orientation change and the received data indicatingthe severity of the impact of the patient, a threshold for inactivity ofthe patient, and determining, based on output from the on-body sensor,that the patient has been inactive for a period of time greater than thedetermined threshold; and wherein determining to capture one or moreimages of the room comprises determining to capture one or more imagesof the room based on the determination that the patient has beeninactive for a period of time greater than the determined threshold. 18.A system comprising: at least one processor; and at least one memorycoupled to the at least one processor having stored thereon instructionswhich, when executed by the at least one processor, causes the at leastone processor to perform operations comprising: monitoring output fromat least one sensor configured to sense, in a room of a building,activity associated with a patient falling; based on the monitoring ofoutput from the at least one sensor, determining to capture one or moreimages of the room; based on the determination to capture one or moreimages of the room, capturing, with a camera positioned to include thepatient within a field of view of the camera, an image of the room;analyzing the captured image of the room to detect a state of thepatient at a time of capturing the image; determining, based on thedetected state of the patient, a potential fall event for the patient;and based on the determination of the potential fall event for thepatient, sending, by a communication device, a message indicating thepotential fall event for the patient, wherein the at least one sensorconfigured to sense activity associated with the patient falling is asensor configured to determine a presence of the patient in the room;and wherein monitoring output from the at least one sensor comprises:receiving, from the sensor configured to determine the presence of thepatient in the room, a signal indicating that the patient is present inthe room, and after a threshold period of time, determining that thepatient has not left the room and that no further signals have beenreceived from the sensor configured to determine the presence of thepatient in the room; and wherein determining to capture one or moreimages of the room comprises determining to capture one or more imagesof the room based on determining that the patient has not left the roomand that no further signals have been received from the sensorconfigured to determine the presence of the patient in the room.
 19. Asystem comprising: at least one processor; and at least one memorycoupled to the at least one processor having stored thereon instructionswhich, when executed by the at least one processor, causes the at leastone processor to perform operations comprising: monitoring output fromat least one sensor configured to sense, in a room of a building,activity associated with a patient falling; based on the monitoring ofoutput from the at least one sensor, determining to capture one or moreimages of the room; based on the determination to capture one or moreimages of the room, capturing, with a camera positioned to include thepatient within a field of view of the camera, an image of the room;analyzing the captured image of the room to detect a state of thepatient at a time of capturing the image; determining, based on thedetected state of the patient, a potential fall event for the patient;and based on the determination of the potential fall event for thepatient, sending, by a communication device, a message indicating thepotential fall event for the patient, wherein analyzing the capturedimage of the room to detect a state of the patient at a time ofcapturing the image comprises: performing image foreground segmentationon the captured image to create a segmented image, performing templatematching on the segmented image to identify a human shape in thesegmented image, and calculating a position and an orientationassociated with the identified human shape in the segmented image, andwherein determining the potential fall event for the patient comprisesdetermining a potential fall event for the patient based on thecalculated position and the calculated orientation.
 20. The system ofclaim 19, wherein determining the potential fall event for the patientbased on the calculated position and the calculated orientationcomprises: monitoring successive image and sensor data after calculatingthe position and the orientation; comparing the successive image andsensor data with prior image and sensor data; determining an activitylevel of the patient based on the comparison of the successive image andsensor data with the prior image and sensor data; classifying thepotential fall event based on the determined activity level of thepatient; and handling reporting for the potential fall event based onthe classification of the potential fall event.
 21. A system comprising:at least one processor; and at least one memory coupled to the at leastone processor having stored thereon instructions which, when executed bythe at least one processor, causes the at least one processor to performoperations comprising: monitoring output from at least one sensorconfigured to sense, in a room of a building, activity associated with apatient falling; based on the monitoring of output from the at least onesensor, determining to capture one or more images of the room; based onthe determination to capture one or more images of the room, capturing,with a camera positioned to include the patient within a field of viewof the camera, an image of the room; analyzing the captured image of theroom to detect a state of the patient at a time of capturing the image;determining, based on the detected state of the patient, a potentialfall event for the patient; based on the determination of the potentialfall event for the patient, sending, by a communication device, amessage indicating the potential fall event for the patient; analyzingthe monitored output from the at least one sensor over a period of timeto determine activities of the patient over the period of time;accessing information indicative of expected activities of the patientover the period of time; comparing the determined activities of thepatient over the period of time to the expected activities of thepatient over the period of time; and based on the comparison revealingthat the determined activities of the patient over the period of time donot match the expected activities of the patient over the period oftime, determining a level of fall risk associated with the patient. 22.The system of claim 21, wherein the operations comprise: determiningthat the level of fall risk associated with the patient exceeds athreshold; and based on the determination that the level of fall riskassociated with the patient exceeds the threshold, sending a message toa monitoring server that is located remotely from the building.
 23. Thesystem of claim 21, wherein the operations comprise: determining thatthe level of fall risk associated with the patient exceeds a threshold;and based on the determination that the level of fall risk associatedwith the patient exceeds the threshold, automatically performing one ormore operations to reduce the level of fall risk associated with thepatient.
 24. A system comprising: at least one processor; and at leastone memory coupled to the at least one processor having stored thereoninstructions which, when executed by the at least one processor, causesthe at least one processor to perform operations comprising: monitoringoutput from at least one sensor configured to sense, in a room of abuilding, activity associated with a patient falling; based on themonitoring of output from the at least one sensor, determining tocapture one or more images of the room; based on the determination tocapture one or more images of the room, capturing, with a camerapositioned to include the patient within a field of view of the camera,an image of the room; analyzing the captured image of the room to detecta state of the patient at a time of capturing the image; determining,based on the detected state of the patient, a potential fall event forthe patient; and based on the determination of the potential fall eventfor the patient, sending, by a communication device, a messageindicating the potential fall event for the patient, wherein sending themessage indicating the potential fall event for the patient comprisessending, to the patient, the message indicating the potential fall eventand providing the patient with an opportunity to cancel the potentialfall event.
 25. The system of claim 24, wherein the operations comprise:determining that the patient has not cancelled the potential fall eventwithin a threshold period of time; and based on determining that thepatient has not cancelled the potential fall event within the thresholdperiod of time, sending a message to a monitoring server indicating thepotential fall event.
 26. The system of claim 24, wherein the operationscomprise: receiving, from the patient, an indication to cancel thepotential fall event; and based on receiving the indication to cancelthe potential fall event, determining an overall activity of the patientbetween detecting the potential fall event and receiving the indicationto cancel the potential fall event.
 27. The system of claim 26, whereinthe operations comprise: determining that the overall activity of thepatient is above a threshold of activity; and based on determining thatthe overall activity of the patient is above the threshold of activity,signaling that the potential fall event was detection of a false fall.28. The system of claim 26, wherein the operations comprise: determiningthat the overall activity of the patient is below a threshold ofactivity; and based on determining that the overall activity of thepatient is below the threshold of activity, determining an orientationof the patient.
 29. The system of claim 28, wherein the operationscomprise: determining that the determined orientation of the patient isupright; and based on determining that the determined orientation of thepatient is upright, signaling that the potential fall event wasdetection of a minor fall.
 30. The system of claim 28, wherein theoperations comprise: determining that the determined orientation of thepatient is not upright; and based on determining that the determinedorientation of the patient is not upright, sending another message tothe patient that provides the patient with another opportunity to cancelthe potential fall event.
 31. The system of claim 30, wherein theoperations comprise: determining that the patient has not cancelled thepotential fall event within a threshold period of time after sendinganother message to the patient that provides the patient with anotheropportunity to cancel the potential fall event; and based on determiningthat the patient has not cancelled the potential fall event within thethreshold period of time after sending another message to the patientthat provides the patient with another opportunity to cancel thepotential fall event, sending a message to a monitoring serverindicating the potential fall event.
 32. The system of claim 30, whereinthe operations comprise: after sending another message to the patientthat provides the patient with another opportunity to cancel thepotential fall event, receiving, from the patient, an indication tocancel the potential fall event; and based on receiving the indicationto cancel the potential fall event, signaling that the potential fallevent was a cancelled fall event.